Students Cheat Less in Online Courses

In "Students Cheat More in Online Courses?", George Watson and James Sottile (Online Journal of Distance Learning Administration, 2010) report that students admit to cheating more in face to face courses than on-line ones. They suggest that this is because the students work together in the classroom and so are included to share answers. To stop the sharing of answers in on-line tests, the authors suggest having the test supervised, or better still change to other forms of assessment, such as assignments. One approach which they surprisingly do not suggest is group work. Students in the face-to-face class are working together should be seen a a positive outcome and could be encouraged by group work. I use online discussion forums as part of assessment, along with assignments.

Assessing for quality decision making at university

ANU Sciences Teaching Building

Greetings from the Australian National University  Science Teaching and Learning Colloquium on Research Led Education, where  by Professor Geoffrey Crisp, is talking on "Assessing for quality decision making at university". He began by pointing out that he had presented on the same spot in 2007, in a demountable building.

As with the 2007 presentation, Professor Crisp started with an image from a medieval painting of a lecture in a monastery. He then showed an image of a modern lecture theatre and suggested not much had changed. e argues that assessment needs to be authentic, being related to real world tasks. He pointed out that a three hour paper based examination is unlikely to relate to what the student needs to know.

Professor Crisp's 2007 talk and his book "e-Assessment Handbook" gave me the courage to decide to give up setting conventional examinations for the university courses I design. What I found surprising is that five years later there would be a need to make this case again. By now the idea that assessment needs to be aligned with the teaching and based on real world tasks should be conventional wisdom and three hour paper examinations relegated to history.

One theme emerging from the event today from several speakers is that learning is partly a socialisation process. University lecturers need to help students learn how to learn, so as to be academics and professionals.

Geoff showed is own  Transforming Assessment website (which I noticed is implemented with Moodle).

Also he recommended the Sbl Interactive scenario software. This was developed at University of Queensland for presenting a scenario to a student  who then makes a decision and the scenario displays the consequences. The software can be used for free if the developed scenarios are shared. I could not find any listed scenarios on ICT. The "Living Without Water" scenario from Engineers Without Boarders shows the potential.

Online Student Readiness Assessment

Open Universities Australia (OUA) offer online tools to help students with Career Advice  and Readiness Assessment for Study.

The readiness assessment asks students to rate themselves using a five point Likert scale (Strongly disagree, Disagree, Neutral, Agree, Strongly agree).  The first eighteen questions, headed "Goals", address the student's attitude to studying online, level of  motivation and time available for study. A second fourteen questions ask if the student has suitable technology for on-line study.

Curiously, one question asks if the student has Windows XP, Windows Vista or Windows 7, or Mac OS X. Apparently students using Linux, Google Android or iOS on an Apple iPad are excluded from study. Having undertaken several online courses, I have had some difficulties using Linux and there would appear to be too small a user base to cater for. However, tablet computer and web appliance operating systems Google Android and Apple iOS would see a large enough potential market.

The student is also asked if they are using a dial-up internet connection and they have a high speed internet connection. Dial-up and 3G connections are apparently considered too slow. I found a 3G wireless connection more than adequate for most online courses (both as a student and tutor). For real time video and audio conferencing careful setting of the system to minimise bandwidth was needed, but most online courses will not require video.

The student is asked if they have a recent version of Microsoft Office. I found that OpenOffice/LibreOffice was an adequate alternative.

The third set of twelve questions asks about the student's approach to study and in particular their writing skills. One problem with the questions is that this is an opinion survey, that is the student is asked if they know what plagiarism is, there is no text of this knowledge administered.

The tool presents the student with an assessment (mine was "Almost Ready", which is curious given I have successfully passed online courses, as well as designing one) and then makes recommendations for first units. In my case PREP03 "Preparing for University Learning Intensive", was recommended. Also Bridging units are offered, which is particularly relevant for those who would not have previously considered university.

It would be interesting to apply this technique to higher degree and research students.

Learning University Teaching: Reflection

Last week I attended day four of an introductory course on university teaching. This was the last day of the course and it was time to reflect on what I learned and suggestions on how to improve the course:

Background to the Course

The course is a non-assessed introduction to university teaching intended for academics just starting their career. It is four days, one day per week, for four weeks, plus some on-line activities using the Moodle Learning Management System. The course is offered to staff by the university's academic education center.

One issue this raises is that all teachers at a university do not start out as "early career academics". Some are "late career professionals", who find they have a taste for teaching. Universities in Canberra, for example, have current and former public servants assisting with teaching. There are also staff of organizations which support the public service. These staff need an introduction to teaching for professionals who have a depth of experience in the workforce, but limited understanding of the academic environment and limited time to acquire it.

Recommendation for Improvements in the Course

The introductory course covers educational theory, but could better use the techniques discussed, to put theory into practice. This would be more efficient, and more credible, demonstrating how the educational theories work:

1. Offer a blended course, with optional face-to-face components: Research presented indicates that having oral presentations is not the most efficient teaching technique. Therefore it would be appropriate to reduce the use of live oral presentations and increase the use of student activities in this course. An appropriate format would be two hours per week optional face-to-face contact (down from five hours mandatory), with the rest of the course on-line.

Also the course covers Equal Opportunity Policy, emphasizing an inclusive approach in both education and employment. One way to do this is to provide flexibility for attendance at courses. Currently the course requires attendance at all four daytime events. These are only offered once per teaching period, at only one location in one city, and only in Australia. Offering the course at multiple times and locations would be prohibitively resource intensive. However, on-line participation could be offered as an alternative to attendance in person.

2. Convert the course materials to an accessible on-line format via the learning management system: Disability Policy is covered in the course. The policy "... incorporates the inclusion of people with disabilities in employment and education to enable them to perform at their best in University life ...". Implementing this policy in the course would be aided by checking that content of paper handouts and screen displays used were large enough for comfortable reading. Also providing the material in alternate formats via the Learning Management System would be useful.

A simple way to provide course materials is with an e-Book, available via the course web site at the beginning of the course. This can contain summaries of what is to be covered, links to readings and the work sheets for activities to be undertaken. The number of readings should be feasible to be read in the time allocated for the course and the class exercises should reference the readings to prompt the students to read them.

4. Add assessment to the course: The course emphasized that students value assessment and the assessment can be used for aiding learning, not just as a test at the end. It would therefore be appropriate to design assessment activities into the course, rather than as an optional extra at the end. At least weekly assessment should be mandatory, along with at least weekly feedback from the tutor to each individual student. The optional credit for "reflection" should be eliminated and replaced with mandatory assessable items.

5. Construct a showcase active learning classroom: It is suggested that the university commission the design and construction of an Active Learning Classroom (ALC) for teaching education techniques. This would be similar to the INSPIRE Centre for ICT Pedagogy, Practice and Research under construction now at the University of Canberra. It would be equipped with computer systems and screens for one large class and for breakout groups. Space for this might be found in the City West Precinct. A design similar to the University of Canberra Teaching and Learning Centre could be used.

5. Design an on-line learning environment: A web site for the course, which conforms to the same standards as other courses, should be developed. This should be tested before the course commences and the design not changed during the course. The web site should be integrated with the face-to-face content.

6. Put assessment first: The course referred to research indicating that assessment is important to students and should be integrated in course design, but this topic is left to last in the course. It is suggested that the topic of assessment be discussed alongside that of preparing course descriptions, early in the course.

My Background

Like many staff I came to university as a Visiting Fellow and ended up doing teaching as an adjunct. We teach material developed in our day-to-day work.

While having provided conventional lectures and examination based teaching for years, I was never comfortable with this mode, as I teach about on-line communication. In 2005 and 2007 I made some ad-hoc attempts at blended learning ("Workshop on the Use of Technology for Museums of the Pacific Islands Region" and "Electronic Document Management, Module 2 of Systems Approach to the Management of Government Information"). Later I was commissioned by the Australian Computer Society to design an e-learning course as part of the ACS Computer Professional Education Program, which is part of a globally accredited postgraduate program.

ACS provided training in techniques for mentored and collaborative e-learning, based on those used at the Open University. I then adapted the same course content for university. The course has worked well, winning an industry award and with one of the students now running it in Canada. My ambition is now to design more such courses, explore the theory behind them and teach others how to do this.

Goals for the Course

My reasons for enrolling in this course were:

1. Cost: The course is free for current staff,
2. Get in before it is compulsory: While there is no requirement for university staff to have training in teaching, this is likely to become more strongly encouraged.
3. Validate vocational training: While I have undertaken a considerable number of vocational short courses, it was useful to check I was up to date with the latest thinking on educational theory for higher education.
4. Help university implement e-learning: While I have been successful at implementing e-learning, my academic colleagues are skeptical of my approach, perhaps due to my not being able to explain it using the correct academic terms to describe it,
5. Ease into postgraduate studies: To see if it would be worth undertaking the certificate in teaching.

Results

1. Completed course: I was able to attend all four days and so was awarded an attendance certificate.
2. Validated vocational training: I was able to verify that my previous vocational training is consistent with university thinking on educational theory.
3. Ready to help advance university teaching practice: I was able to see that the university was striving to implement blended and e-learning.
4. Applying for the certificate in teaching: I have applied to study the certificate in teaching.

Learning University Teaching: Lesson 4

Last week I attended day three of an introductory coruse in university teaching. The fourth and last day this week is about "Designing and Marking Assessment Tasks". Here is my thoughts in preparation:

Designing and Marking Assessment Tasks

It is useful to see what the public perception of university assessment is, or at least the media's perception. So here are the recent five top stories from Google News which mention "University Assessment":

1. Two Mumbai varsity staffers caught stealing answer papers, Hindustan Times,23 Jun 2011‎: The University of Mumbai on Wednesday caught two of its temporary staff trying to steal engineering answer papers from the university's central assessment centre at Kalina. The duo had tied eight answer papers to their legs and was walking out when the ...
2. Assessment and learning in the digital age, Media Newswire (press release): The symposium, Assessment and learning in the digital age, will take place at the University of Bristol's Graduate School of Education ( GSOE) on Friday 17 June from 1 to 5 pm in Room 4.10, GSOE, 35 Berkeley Square, Bristol. ...
3. KCPE and KCSE face scrapping, The Standard, Augustine Oduor, ‎Jun 21, 2011‎: The exams whose hallmark has been cutthroat competition among schools, and which hold the key to good secondary places and lucrative university courses, might be replaced with a list of assessment tests spread across the learning system. ...
4. Teaching quality under pressure as unis chase money, The Australian, Julie Hare, ‎Jun 20, 2011‎: SPIRALLING class sizes, overcrowding, tutorials replaced by seminars, few avenues for feedback and interaction, a shift to online and peer-assessment as a cost saving measure -- the dire state of teaching in Australian universities emerges from just a ...

These indicate that assessment issues are of concern globally, including: fairness, adaption to on-line delivery, reduction in the use of large end of semester examinations and peer assessment.

A search of Google Scholar, shows six documents featuring the words "university assessment" in the title for 2011:

1. Talking the talk: oracy demands in first year university assessment tasks: C Doherty, M Kettle, L May… - Assessment in Education: …, 2011 - informaworld.com... 18, No. 1, February 2011, 27–39 ISSN 0969-594X print/ISSN 1465-329X
2. Towards Fairer University Assessment: Recognising the
   
Concerns of Students N Flint… - 2011 - books.google.com: After all the hours of studying, reading and preparation, the nights spent revising and the writing and re-writing of assignments, 'success' for university students can often be represented with a single grade or digit, ...
3. 'Worldmarks': Web guidelines for socially and culturally responsive assessment in university classrooms, CE Manathunga, D MacKinnon - … Conference 2002: The …, 2011 - espace.library.uq.edu.au ... understanding. Yet university assessment in Australia is often based on a western template of knowledge, which automatically places International, Indigenous, as well as certain groups of local students at a study disadvantage. ...
4. 'In Press' Measuring up? Assessment and students with disabilities in the modern university, J Bessant - International Journal of Inclusive Education, 2011 - researchbank.rmit.edu.au, International Journal of Inclusive Education, vol. TBA, no. TBA, pp. ...
5. Teaching in the Corporate University: Assessment as a Labor Issue, J Champagne - 2011 - academicfreedomjournal.org ... Following the self-study, our provost established the Coordinating Committee on University Assessment. In March 2006, that ...
6. Towards fairer university assessment: recognizing the concerns of students,: A Iredale - 2011 - eprints.hud.ac.uk: This book is aimed at higher education academics, administrators and managers, researchers, and to some extent undergraduate and postgraduate students. It explores assessment as a determiner of student satisfaction, and is based upon Nerilee Flint's PhD thesis. A ...

These articles address fairness, cultural responsiveness and dealing with disability in assessment. Most interesting is that two of the six refer to the book: "Towards Fairer University Assessment: Recognizing the Concerns of Students" by Nerilee Flint (Routledge, 2011). Dr Nerilee Flint is Education Advisor, Student Equity, ANU. In the paper "Unfairness in educational assessment: Modifiers that influence the response students have to a perception of unfair" (2007) and later in her book, she suggests assessment is important to universities and is a powerful way to influence student behavior.

Integrating Assessment with Course Design

While education theory and items in the media suggest assessment is important, in practice it tends to be left to later, both in design and delivery of courses. Design of assessment is generally left until after course content is decided. Also much of the assessment of a university course is by way of an end of course examination, where the results of that examination cannot be used to help the student with learning in that course (as the course is over). If assessment is important, then it should be designed alongside the content and delivered before the end of the course.

My approach is to provide all assessment items at the beginning of the course (or preferably before the student enrolls). As an example, all assessment items for my two e-leaning courses "Green Technology Strategies" and "Electronic Document and Records Management" are available before the student starts the course. The assessment items are based on real world tasks the student will be expected to be able to carry out after the course. This avoids the philosophical conundrum of attempting to assess what the student "knows", instead assessing what they can do. It also appeals to students looking to do the course in order to get a better job.

To promote a sense of fairness and to avoid unnecessary requests for remarking, when marking assignments I first make detailed comments, giving the students examples of what is good, what could be improved, how and why. Rather than add up some marking scheme to give an arbitrary total, I instead form an assessment of the grade of the work (fail, pass, credit, ...) and then a mark within that grade. I provide the student with the detailed comments, the grade and the mark. This is a way to clearly tell the student my assessment of their work (this "credit" level work). It avoids the time wasting scramble for marks, where the student is tempted to ask for a few more marks to push them up into the next grade.

What is assessment for?

The accepted wisdom in educational theory is that there is formative and summative assessment. Formative assessment helps the guide the student with their learning while it is in progress, whereas summative assessment is for an external report on the results at the end. While, Harlen & M. James ("Assessment and Learning: differences and relationships between formative and summative assessment", 1997) argue that things are not as clear cut in the real world and there is a creep to wards using assessment originally developed for formative use as summative, this still seems a useful distinction to make. However, I like to provide marks for formative work, as a way to motivate students to do it (24% of the total marks seems to be sufficient).

When deciding how much assessment and in what form to use in an ANU course, I did a quick survey of Australian university assesment. For the usual 13 week course (of 9 to 10 hours work per week), universities typically require 40 to 60 words per percent. That is 4,000 to 6,000 words of assignments written by students for a complete course, with a set number of words corresponding to a length of examination or oral presentation. As an example, University of Melbourne equates one hour of examination or ten minutes of individual oral presentation to 1000 words of assignment.

Automated Assessment

Learning Management Systems, such as Moodle, have provision for automated quizzes built in. This would seem to be ideal for formative assessed, particularly when assessing what the student already knows at the start of a course, so they can be guided to concentrate on what they don't know. Professor Geoffrey Crisp, Director of the Centre for Learning and Professional Development, at University of Adelaide and author of the "e-Assessment Handbook" has given seminars in Canberra on how to automate assessment. This could be useful, but it takes considerable work to set up in advance and so far none of the organizations who commission me to design cruses have been willing to fund the work needed for this.

In designing automated assessment, it needs to be kept in mind that the system must, if possible, be designed to be usable by a wide range of students. As an example, the student may be remote from the campus, on a slow telecommunications link, have a low powered computer or have a disability. This needs to be taken into account when designing the assessment. As an example, if the assessment is in the form of a Flash animation, the student may not be able to use it, thus be disadvantaged and leaving the institution open to charges of unlawful discrimination.

In addition assessment should not be designed to make up for inappropriate course design. As an example, it is not possible to provide individual feedback in a live lecture to hundreds of students. "Clickers" can be used to conduct a quick quiz, students can submit questions via pieces of paper or Twitter. But one lecturer can't deal with all the questions from hundreds of students in a live environment and should not pretend they can.

Making Course Assessment Palatable for Students and Teachers

New ABU staff member, Dr Nerilee Flint, Education Advisor, Student Equity, presented a talk on "Assessment--making it fair" at the Australian National University, 13 May 2011. This was based on her PHD research, now published in a new book "Towards Fairer University Assessment: Recognizing the Concerns of Students" (Routledge, 2011).

Nerilee has a background as a teacher, who then moved to the university sector. She used "grounded theory methodology" for her work. After extensive research she concluded the central issue with student's view of assessment was frustration. This frustration comes from different views of assessment by students and teachers and between different students.

Several examples of areas for misunderstanding and frustration were given. The first was assignment deadlines which have nor rational and re not consistently enforced. Another example was vague definitions of word counts for work. These seemed to me not to be fundamental issues with assessment, but simply examples of poor assessment design. Two ways to overcome this were better training for individual teachers and the sue of standards for assessment within and between institutions.

Nerilee presented a theoretical framework showing what the student takes into account in their response to assessment. One comment was that some students will check what the assessment is before enrolling and use this to help decide which course to do. To me this seems a statement of the obvious. As a student I avoid courses which have an semester examination as the primary form of assessment (also as a teacher I think this is not good for learning). But apparently many course designers do not realise that assessment is very important to students when choosing courses.

Factors in student's view of assessment Nerilee found included the teacher's perceived skill in teaching and the extent to which they are seen to be caring about students. One example is a tutor who does not moderate the mature age students contribution, so the younger ones get a chance to speak. This is not an issue for my tutorials, which are online and so everyone can talk (and behind the scenes I provide the hesitant students with individual encouragement to post). Also I provide an individual response to every student every week, which shows the tutor is taking an interest in the student as an individual.

One useful reminder was that not all students understand what the assessment process is. In my view the use of templates and particularly a Learning Management System (such as Moodle) can help with this. With an ad-hoc paper based approach, each teacher prepares their own assessment items and so it is easy to have inconsistency across an institution. If templates are used with standard wording this is much more difficult. Also if an LMS is used, which has links to the same assessment guidelines on every course, this can help.

While Nerilee concentrated on the student's view of assessment, the suggestions presented could equally help assessment more palatable to teachers. Assessment is mostly seen as a drain on time and resources by teachers, who would like to get on with the real teaching. In my view this is a misguided approach. Assessment is an essential part of learning (in one case I enrolled in a course with no assessment and found it frustrating). By integrating the assessment in the way they teach, staff can reduce their workload and also remove many of the day to day annoyances of students grumbling about assessment problems.

Before Nerilee's presentation, Professor John Dearn, talked about the new policy on assessment he is developing for ANU. It is unfortunate that this did not follow Nerilee's talk, as that would have informed the issues.

Professor Dearn mentioned the university had dozens of policies, mentioning "assessment" (I found 254). He discussed the difficulties with formulating a policy for an institution, with differences between disciplines on what assessment is and the link between policy and practice. In my view, one way a univerity could better link policy and practice would be by integrating the teaching of teaching and learning. The unviersity currently has a separate units dealing with teaching teachers, helping students with learning and provision of learning technology. Institutions such as the University of South Australia (which I visited last week) have a more integrated approach.

Professor Dearn then listed some issues:

1. Hard to write learning outcomes and align them with teaching strategies: Students need to see a link between the assessment and what the course is about. Professor Dearn mentioned that students get cynical where the assessment is an examination at the end. To me the solution to this is obvious: stop putting so much assessment at the end of courses and use forms of assessment based on simulations of real world experience, not paper based tests. I now use assessment every week in my courses (starting from week one) and have mostly given up using end of semester examinations. Instead I get the students to do what they are being trained to do and asses that.
2. Feedback for formative assessment: Students complain about the lack of feedback. Professor Dearn said that staff are concerned that early feedback in a course increases the workload. To me the answer is obvious: include regular small assessment items which are easy to mark but do contribute to the final result. The staff time needed to do this can be provided by eliminating traditional lectures, which have been shown not to be useful for learning, are not popular with students and waste resources.
3. Blind marking: It is suggested marking should be blind, so that staff do not know which student's work is being marked. This should be reasonably easy to do using a Learning Management System, where submissions can be made electronically and the system can keep track of which assignment is from who, without showing that information to the assessor. An interesting issue not raised by Professor Dearn is if assessment should be double blind: that is should the student not know which staff member did the marking, so that the staff can give full and frank comments (as is done with reviewing of academic papers for publication).
4. Marks for attendance: The university currently allows marks for attendance. Professor Dearn appeared to have concerns about this practice,which in my view this is not a good idea. There is a very simple alternative: provide a small mark for a small assessment item carried out after a session. The student is then not marked for attendance, but on what they learned from the attendance. This encourages the students to attend, and also helps them identify what they have to work more on. This can be easily done with the LMS and I use it routinely.
5. Normative marking: Should students be assessed on objective criteria, not according to a marking distribution for the class? In my view a bit of both can be used. Assessment can be done objectively and then double checked to see how the student does against others in that class, the same student in other classes and students in the past. The University's Research School of Computer Science has a sophisticated locally developed computer system, which is used by examination committees to compare courses and students, to ensure consistency of marking. This is a process I was skeptical of (and a little afraid of) until I participated in it and saw the system in use in a collegiate environment. This is something which could be added to the LMS to improve results.

Why consider on-line assessment?

In 2007 I attended a seminar on e-assessment by Professor Geoffrey Crisp, read his excellent book "e-Assessment Handbook" and in early 2008 a workshop on e-assessment. I now routinely use e-assessment, having given up on written examinations. However, I still use a relatively conventional adaptation of assignments and tutorial participation, simply replacing the paper assignments and face to face tutorials with online equivalents. Perhaps it is time to go back and ask: "Why consider on-line assessment?". This and other questions are asked in "On-line assessment" (2002) from the "Assessing Learning in Australian Universities", at the Centre for Students of Higher Education at University of Melbourne.

My last examination

Today I told my fellow educators for the course COMP3410 at the Australian National University that this was my last examination. What I did not expand on was that this was not just for this course, I will not set any examination questions for any course. My hope is to able to continue to contribute to courses at ANU and elsewhere, and to design assessment for them, but do not ask me to write examination questions.

Last year I decided to give My Last Lecture and put into practice what I have been learning about combining online teaching and live group work. This was not an easy decision and has taken considerable work, both in reformulating course material and explaining the change to my colleagues.

My first ANU course with no lectures and no examinations finishes in a few weeks time and so far has gone reasonably well. Reformulating the material did take some work. What surprised me was the amount of explanation I had to do to colleagues. Many had difficulty with the idea of a course with no lectures. Even after I had announced that I would not be giving any more lectures there were many who asked me to give lectures. It took some time to reassure them that there were workable alternatives to lectures and this was permitted under university rules.

What I found surprising was that the formal university processes had no problems with a course with no lectures or examinations. Provided what was being taught and how it was being assessed was clearly spelt out, there was no impediment to a course without lectures or examinations.

The group who did not need any convincing were the students, who were very happy with the idea of no lectures. Some students seemed to think no lectures meant no work and an easy course, but soon changed their view when they saw there was compulsory assessed tutorial work each week. Some dropped out early, but not as many as for a conventional course.

I don't like setting examinations to be conducted on paper, over several hours, without the student being able to use external resources. This is not an effective form of assessment, nor is it useful for learning.

An examination is very different to the ways the students could expect to use what they are learning the real world. What is more realistic is have them briefly reply to a few short questions, discuss issues in a group and to write reports over days or weeks, with a library and the Internet at their disposal. That is the form of assessment I have used in the Green ICT course, with assessment for weekly discussion and written assignments.

There is a role for tests in courses and this can be by interactive assessment. These can be used to asses the student's level of knowledge to help with the learning as well as to give them a final grade. But the assessment should be a realistic facsimile of what happens in the real world. Assessment can use simulations and other techniques for being more realistic. Tests can be online and in part multiple choice. There can be many short tests, rather than a few large ones. I hope to explore this next year.

Assessment and Teaching of 21st Century Skills project

Assessment and Teaching of 21st Century Skills project aims to develop globally standardised ICT-based assessment for school education which emphasise the use of ICT. The project came from a Call to Action by Cisco, Intel and Microsoft in January 2009. An example used in the call has students using the web to help solve a problem. What will be a bit disconcerting for education traditionalists is the spelling mistake "poplation" in the example.

It would be easy to dismiss ATC21S as self interest by computer companies wanting to sell computer hardware and software to schools. What might be reassuring (or not) is that the project is headed by Professor Barry McGaw at the University of Melbourne, with other academics from the University of California, Berkeley, University of Szeged, Hungary, University of Washington,and University of Toronto.

ATC21S has Working Groups on:

1. 21st Century Skills
2. Methodological Issues
3. Technological Issues
4. Classroom Learning Environments and Formative Evaluation
5. Frameworks for New Assessments

Interactive Presentation on Carbon Footprint

A "flowgram" interactive "Presentation on Carbon Footprint Project-Basic Understanding" is available. This uses the
Adobe Flash Player to provide audio synchronised with web based content in the web browser.

Unlike some video based tools, Flowgram shows the actual web page in a frame in the web browser, not just an image captured of the page. In this case the narrator describes carbon footprint while pages from the US EPA are displayed. Text from the web page is highlighted as the narration continues. You can pause the presentation, rewinf and fast forward like a video.

You can also click on links in the displayed web page to break out of the confines of the prepared presentation. However, if you click to go elsewhere it is difficult to find your way back to the presentation. When you do get back to the presentation, it starts from the beginning, not where you left off.

e-Assessment makes the grade

The University of Canberra is having a Learning & Teaching Week, 9-12 September, 2008 with free events, which non-UoC people, such as myself were invited to. One of the last sessions was the seminar "e-Assessment: More than just a grade", with Professor Geoffrey Crisp, Director of the Centre for Learning and Professional Development, University of Adelaide and author of the "e-Assessment Handbook".

I attended a similar seminar by Professor Crisp, at ANU last October. The presentation has been updated with new material since then. He showed simple ways to implement computer assisted assessment using existing tools such as the Moodle and Web CT learning management systems, in conjunction with web based simulations and other interactive applications. There are demonstrations of the techniques available online create your own account for acceess). He also demonstrated the use of Ken Taylor's world leading VotaPedia system, which we used at the Open 2020 Summit.

This was a very useful and skillfully presented seminar, which combined high level educational theory, painlessly combined with practical tips for teaching. About the only negative was that the University of Canberra's technology was not quite up to the job, with the system controlling the video projector crashing and other audio visual problems impeding the presentation. However, Professor Crisp was able to skillfully work around these and use them as lessons about e-learning.

e-Assessment: more than just a grade This session explores some of the opportunities offered by online assessment to improve student outcomes and the quality of the assessment tasks. The work is based on the Carrick Associate Fellowship project and describes how academics can prepare interactive, computer-based assessments using helper tools such as browser plugins, java applets, QuickTime VR and Flash. The aim of the project is to assist teachers to move beyond simple multiple choice questions in an online environment to provide much richer, authentic and meaningful assessment tasks for students.

Interactive e-Assesment

Professor Geoffrey Crisp, Director of the Centre for Learning and Professional Development and Professional Development and Director of Online Education at the University of Adelaide, presented a seminar on "Assessment: more than just multiple-choice" at ANU in Canberra today. He showed a simple way to implement computer assisted assessment using existing tools, supported by research. I have been doing some of this with Moodle for commercial courses, but did not consider to be academically respectable. Now I know it is okay, I can use it for university courses. ;-)

Geoff started with an image from a medieval painting of a lecture in a monastery. He pointed out that apart from the lack of women, much of the way a lecture was presented then is much the same as now. The lecturer was at a podium at the front of the room, the students were in rows, some listening, some reading, some talking.

Geoff went through the types and purposes of assessment (it is not just because you need to pass). One use is to see what the students already know. "Clickers" can be used to get a quick assessment of the students knowledge in class. But more complex electronically collected assessment need not be marked automatically, it can be used just to collect the answers for human marking. However, before setting assessment the teacher needs to decide what scheme is appropriate and what is being assessed.

e-Assessment can have advantages for immersion and interactivity, social interactions. The results can look like computer and role playing games.

There are some examples of research on interactive, some beter than others, such as Literature Review of E-assessment (Futurelabs) and "Effective Practice with e-Assessment" (JSC).

Geoff showed an approach where the interaction is separated from the assessment. In this model the question invokes the interactive component. As an example the question is asked in a web page in a learning system, such as Moodle, this then invokes an interactive application provided via a web browser plugin and when the plugin is finished the student is returned to the web page where they can enter their answer.

What I found confusing was that this did not seem to be "interactive assessment", in that there was no interaction between the system asking the question and the application the student uses to research the answer. The computer presents a tradition text question, launches the interactive plugin for the student to interact with and then asks them for a traditional answer. There no assessment in the interactive environment. Geoff pointed out that this makes it a lot easier to create and is more flexible, but it took me some minutes to work out that this could be considered a legitimate "interactive" assessment.

Geoff pointed out the advantages of this approach: the academic can use pre-prepared modules and ask different questions about it for different levels of students. What also appears an advantage is that this only a small step beyond the computer based tutorials I prepare for ANU courses and is essentially the same as the system I used for a commercial training course.

Interactive Tutorial Example

With the ANU tutorials, a web page is prepared which looks much like a traditional set of notes for a tutorial/lab. However, the student is provided with a hypertext link which takes them to an online tool which they use to carry out part of the work. They copy some parameters from the tutorial sheet into the tool and copy some results back as their answer. An example is questions one to three of the metadata tutorial for "IT in E-Commerce" (COMP3410/COMP6341). In this the student is first asked to hand code some metadata, then use an online tool and compare the results and reflect on the differences. It would not have occurred to me to describe this approach as "interactive".

Interactive Assessment Example

With "Writing for the Web", a one day course prepared for a local government agency, there are a series of modules consisting of a set of slides used by the presenter live in the classroom, with accompanying notes for the students to read, followed by an exercise for them to do. The slides, notes and exercises are implemented using Moodle. The assignment module of Moodle was used to implement the exercises, but with no expectation that the results the students, just as a way for the student to type in the answers.

As an example in "Search the web for useful words", the student is asked to conduct search using two web tools and compare the results. They type their answers into the Moodle assignment web form (only visible to students enrolled in the course).

Interactive Assessment for Public Service Course

The Electronic Document Management module I am preparing for Systems Approach to the Management of Government Information was going to use the same style of online exercises as the writing fore the web course. However, I had assumed that I would need to provide a paper based test as well. But Professor Crisp's seminar seems to show that the interactive assessment could be used at least for part of the assessment. What would also make this easier would be purpose built flexible learning rooms for this form of teaching.

Postgraduate Online Professional Education for ICT

The ACS provides online courses in IT management using a Moodle based system for its Computer Professional Education Program. ACS also provide a Diploma of Information Technology for budding IT professionals in the Asia-Pacific region. e-Assessment could be of value for these.

More Interactive Assessment

However, to me having a web page launch a web application and then ask the student to copy result from it into a web form is not really "interactive assessment". Geoff commented that the assessment can be built into the interactive application, but this then reduces the flexibility. However, a level of interactive could be created by the assessment system passing parameters to the application and the application passing back some results at the end. This could use Web Services. It could be as simple as the assessment system adding some parameters to the end of the URL used to invoke the application. Results could be returned using XML encoded data, as is done for electronic tax returns, or even just a web page with the results included with Microformats. This would need some work, as my one attempt at including microformats in Moodle, the Moodle web editor modified the content breaking the Microformat.

Next Gen Not Wired

Geoff ended the seminar by cautioning that "Next Gen" students will not necessary benefit from, or want to use, social networking tools for education, even thought they use them socially. So educators should not assume the students will want to do everything via a computer, mobile phone and iPod.

See Also

• Centre for Learning and Professional Development's moodle system.
• Books on e-Learning
• "The e-Assessment Handbook" by Geoffrey Crisp, Continuum International Publishing Group, 2007 (available from Amazon.Com):

  Assessing learning in an online environment is being used by teachers and institutions at an increasing rate. Learners are demanding a more flexible approach to assessment activities just as they have done with learning. This demand will increase as online practice becomes embedded into all schools, further and higher educational courses and corporate training programs. As students are engaging with content in an online environment, they will also need to be assessed using the same tools that are used for the learning experience. This book provides practical guidance to various aspects of online assessment including:

  • types of assessment
  • choosing the right software
  • examples of e-assessment over a wide-variety of disciplines
  • making e-assessment interactive

  
  
  Table Of Contents Preface
  Section 1 How does assessment affect learning?
  1. Overview

  • Why do we assess?
  • Relationship between learning and assessment
  • Assessment for learning
  • Assessment and grading

  
  
  2. Guidelines for assessment practices
  - Professional bodies publishing guidelines on assessment
  - Assessment models
  - e-Assessment practices and their relationship to learning
  3. Types of e-Assessment items
  - types of e-assessment
  4. Cosequences of using e-assessment
  - Risks involved in using e-assessment
  - How is e-assessment currently being used?
  - What are the limitations of e-assessments?
  5. Choosing the right e-assessment software
  - choosing an appropriate assessment format
  - how do you want to administer your e-assessments?
  - what do you want to administer your e-assessments?
  - what type of questions do you want to ask?
  - choices in e-assessment software
  - other types in online assessment software
  - adaptive assessment software
  
  Section 3 What does e-assessment look like?
  6. e-assessment examples by discipline
  - General sources of e-assessment questions
  - Accounting, Commerce and Economics
  - Arts and Humanities
  - Biological and Health Sciences
  - Chemical Sciences
  - Earth and Environmental Sciences
  - Physics and Engineering
  - Mathematics
  
  Section 4. What are the practical issues when doing e-assessment
  7. Item and Test Bank use
  - constructing Ttem and Test Banks
  - using Item and Test Banks
  - non-adaptive test delivery methods
  - adaptive test delivery methods
  8. Grading and marking Items from the Test Bank
  - general grading and marking issues
  - negative and confidence level marking of objective items
  - marking selective response items
  9. Validity and reliability
  - overview of validity and reliability
  - classical test theory
  - Item response theory
  - Rasch modelling
  - estimating reliability
  - evaluation of test banks
  

  10. Standards, specifications and guidelines
  - assessment standards
  - national standards
  - professional association standards and guidelines
  - identifying standards
  - content standards, specifications and guidelines
  - assessment content metadata
  - interoperability
  11. Security issues
  - authentication of appropriate access
  - secure assessment environments and software
  - standards and guidelines
  - what to do when things go wrong
  12. Accessibility issues
  - statutory requirements for accessibility
  - good practice guidelines

  - testing accessibility

  - improving accessibility

  - assistive technologies

  Section 5 How Do I Make e-Assessment Interactive

  13. Assessing discussion groups and collaborative tasks

  - communicating and collaborating online

  - overview

  - examples of software for e-collaboration

  - design principles

  - collaborative problem solving and group work

  - online discussion groups

  14. Interactive assessment and Java applets

  - learning designs

  - simulations

  - Java applets

  15. Assessing role playing and games

  - characteristics of online role-plays

  - senario-based

  - game-based

  Section 6 What about the future?

  16. The future of e-assessment

  - current perceptions of where e-assessment is heading

  - drivers for change

  - what will it be like in the future

  Appendices

  Glossary